Adaptive imaging using digital light processing

ABSTRACT

A system ( 100 ) for the adaptive imaging of a scene includes a digital light processing apparatus ( 150 ) adapted for controllably reflecting an image of the scene in at least a first direction to thereby reflect an intensity modulated image of the scene along at least the first direction, an image detector ( 140 ) for detecting the intensity modulated image of the scene and generating corresponding image data, and an image data processor ( 154 ) for processing the image data into control data and providing the control data to the digital light processing ( 150 ) apparatus for control thereof.

CROSS-REFERENCE TO RELATED APPLICATIONS

This application is based on U.S. provisional patent application Ser.No. 60/311,158 filed Aug. 9, 2001, which is incorporated herein byreference for all purposes and from which priority is claimed.

BACKGROUND OF THE INVENTION

1. Technical Field

The present invention relates to techniques for digitally capturing andprocessing high dynamic range still and video images of a scene usingrelatively low resolution image detectors, and more particularly, totechniques capable of adaptively capturing and processing such images.

2. Background Art

In digital photography and video imaging, it is often necessary tocapture images of a scene which includes areas of greatly disparateluminance. An extreme example lies in the imaging of a solar eclipse,where a nearly black object, the moon, is directly adjacent anextraordinarily bright object, the sun. Other more common examplesinclude photographing a person with the sun in the background,videotaping a sporting event in the late afternoon hours where shadowsare evident, and capturing images of a fireworks display.

In each of these examples, the range of luminance values needed to fullycapture the scene is high, ranging from near black to extremely bright.As a typical image capture device, such as a charge-coupled device(“CCD”) operates with eight bits of resolution, only 256 luminancevalues are available to span the entire brightness spectrum in suchcircumstances, and often yield a poor images. While higher dynamic rangeimage sensors, such as sixteen bit CCDs, are commercially available,they come at an increased cost which may be undesirable, and regardlessof scene brightness operate within a fixed dynamic range.

Accordingly, there have been several attempts to provide a technique forcapturing and processing high dynamic range still and video images of ascene using relatively low resolution image detectors, such as theabove-mentioned eight bit CCD. Those attempts may be generally groupedinto sequential exposure change methods, techniques using multiple imagedetectors, approaches using multiple sensor elements within a pixel,adaptive pixel exposure techniques and methods employing spatiallyvarying pixel exposures.

The most obvious approach is to sequentially capture multiple images ofthe same scene using different exposures. The exposure for each image iscontrolled by either varying the F-number of the imaging optics or theexposure time of the image detector. A high exposure image will besaturated in the bright scene areas but capture the dark regions well.In contrast, a low exposure image will have less saturation in brightregions but end up being too dark and noisy in the dark areas. Thecomplementary nature of these images allows one to combine them into asingle high dynamic range. Such an approach has been employed inJapanese Patent No. 08-223491 of H. Doi, et al. entitled “Image Sensor”(1986) and by others since then. The sequential exposure charge approachwas subsequently extended by using acquired images to compute aradiometric response function of the imaging system, e.g., as describedin T. Mitsunaga et al., “Radiometric Self Calibration,” 1 Proc. ofComputer Vision and Pattern Recognition '99,” pp. 374-380 (1999).

Unfortunately, sequential exposure change techniques are inherentlysuited only to static scenes, and the imaging system, scene objects andtheir respective radiance levels must remain constant during thesequential capture of images under different exposures. If the imagescan be captured in quick succession, they can be merged to obtained highdynamic range images at a reasonable frame-rate, as described in U.S.Pat. No. 5,144,442 to R. Ginosar et al. However, the above notedconstants must be maintained.

The stationary scene restriction faced by sequential capture may beremedied by using multiple imaging systems. This approach is describedin the above mentioned Japanese Patent No. 08-223491, as well as inJapanese Patent 08-340486 to K. Saito (1996) and elsewhere. Beamsplitters are used to generate multiple copies of the optical image ofthe scene. Each copy is detected by an image detector whose exposure ispreset by using an optical attenuator or by changing the exposure timeof the detector. While this approach is capable of producing highdynamic range images in real time, with both scene objects and theimaging system free to move during the capture process, a disadvantageis that this approach is expensive as it requires multiple imagedetectors, precision optics for the alignment of all the acquired imagesand additional hardware for the capture and processing of multipleimages.

Another approach to high dynamic range imaging uses a different CCDdesign, where multiple sensor elements lie within a pixel. In thisapproach, each detector cell includes two sensing elements, i.e.,potential wells, of different sizes and hence sensitivity. When thedetector is exposed to the scene, two measurements are made within eachcell and they are combined on-chip before the image is read out. Such anapproach is proposed in U.S. Pat. No. 5,789,737 to R. A. Street, andelsewhere. However, this technique is also expensive as it requires asophisticated detector to be fabricated. In addition, spatial resolutionis reduced by a factor of two since the two potential wells take up thesame space as two pixels in a conventional image detector. Further, thetechnique is forced to use a simple combining technique for the outputsof the two wells as it is done on-chip.

A further approach to high dynamic range imaging has been proposed in V.Brajovic et al., “A Sorting Image Sensor: An Example of MassivelyParallel Intensity-to-Time Processing for Low-Latency ComputationalSensors,” Proc. of IEEE Conference on Robotics and Automation, pp.1638-1643 (1996). There, a particular solid state image sensor isdeveloped where each pixel on the device includes a computationalelement that measures the time it takes to attain full potential wellcapacity. Since the full-well capacity is the same for all pixels, thetime to achieve it is proportional to image radiance. The recorded timevalues are read out and converted to a high dynamic range image. Thisapproach faces the challenge of scaling to high resolution while keepingfabrication costs under control. In addition, since exposure times canbe large in dark scene regions, the method is expected to be moresusceptible to motion blur.

Finally, an approach to high dynamic range imaging using spatialvariation of pixel sensitivity has been disclosed in S. K. Nayar et al.,“High dynamic range imaging: Spatially varying Pixel exposures,” Proc.of Computer Vision and Pattern Recognition '00 (2000). Different (fixed)sensitivities are assigned to neighboring pixels on the image detector.An important feature here is the simultaneous sampling of spatialdimensions as well as the exposure dimension of image radiance: when apixel is saturated in the acquired image, it is likely to have aneighbor that is not, and when a pixel produces zero brightness, it islikely to have a neighbor that produces non-zero brightness.Unfortunately, a trade-off of spatial resolution of an image is made inorder to improve brightness resolution, or dynamic range. Moreover, thefixed pixel exposure values do not provide the ability to measure eachimage brightness with the best precision possible. Accordingly, thereremains a need for an imaging technique which optimizes the dynamicrange of a captured image of a scene, regardless of the dynamic range ofthe image sensor employed.

SUMMARY OF THE INVENTION

An object of the present invention is to provide an techniques fordigitally capturing and processing high dynamic range still and videoimages of a scene using a relatively low dynamic range image detector.

Another object of the present invention is to provide techniques fordigitally capturing and processing high dynamic range still and videoimages of a scene at reduced costs.

A further object of the present invention is to provide techniquescapable of adaptively capturing and processing high dynamic range stilland video images of a scene, regardless of the dynamic range of theimage sensor employed.

Yet another object of the present invention is to provide techniquescapable of adaptively capturing and processing high dynamic range stilland video images of a scene using feedback from an image sensor tooptimize such image capture and processing.

In order to meet these and other objects of the present invention whichwill become apparent with reference to further disclosure set forthbelow, the present invention discloses systems and methods for theadaptive imaging of a scene. In one exemplary embodiment, the systemincludes a controllable digital light processing apparatus adapted forreflecting an image of at least a portion of the scene in at least afirst direction to thereby reflect an intensity modulated image of thescene along at least the first direction, an image detector fordetecting the intensity modulated image of the scene and generatingcorresponding image data, and an image data processor for processing theimage data into control data and providing the control data to thedigital light processing apparatus.

In a preferred arrangement, the system also includes an image focusingdevice, e.g., a lens, positioned between the scene and the digital lightprocessing apparatus, for focusing an image of the scene onto thedigital light processing apparatus. A second lens may also be positionedbetween the image detector and the digital light processing apparatus.

In a highly preferred arrangement, the controllable digital lightprocessing apparatus is a digital micromirror device, and the imagedetector is a CCD camera. The image data processor may be either ageneral purpose computer including software, or take on a hardwareimplementation. Advantageously, the system may include a second adigital image detector positioned to receive a modulated image from thedigital light processing apparatus along a second direction ofreflection. The system may also include a beam splitter for splittingthe modulated image and two digital image detectors, or alternatively, asecond controllable digital light processing apparatus to generate afurther modulated image of the scene. Thus, systems with cascadingdigital micromirror devices are presented.

In an alternative embodiment, the present invention provides a systemfor adaptively imaging a scene including a controllable digital lightprocessing apparatus, image detector, an image processing apparatus, andan image focusing device. The image focusing device is positionedbetween the scene and the digital light processing apparatus, andfocuses an image of the scene onto the controllable digital lightprocessing apparatus. In this arrangement, the image data processorprocesses a detected image with filter data to generate final image datafrom the detected image data. Preferably, such processing involves theconvolution of raw image data with filter data.

The present invention also provides methods for controlling the adaptiveimaging of a scene using an N-level digital light processing apparatusadapted for controllably reflecting an image of at least a portion ofthe scene in at least one direction to thereby reflect an intensitymodulated image of the scene in that direction. One exemplary methodincludes the steps of (a) setting a digital light processing apparatuscontrol signal corresponding to a portion of the scene to an initiallevel N; (b) taking a first measurement of scene energy at one or morepoints corresponding to the scene portion using the initial controlsignal; (c) revising the control signal to a second level using thefirst measurement; (d) taking a second measurement using the revisedcontrol signal; (e) determining whether the second measurement is belowa threshold value; (f) if it is, updating the control signal to a thirdlevel using the second measurement; and (g) taking a third measurementusing said updated control signal.

The accompanying drawings, which are incorporated and constitute part ofthis disclosure, illustrate preferred embodiments of the invention andserve to explain the principles of the invention.

BRIEF DESCRIPTION OF THE DRAWINGS

FIG. 1 is a functional diagram of an exemplary system in accordance withthe present invention;

FIG. 2 is an illustrative diagram showing the reflection of light raysin a digital micromirror device;

FIG. 3 is a functional diagram of an exemplary system in accordance withone exemplary embodiment of the present invention;

FIG. 4 is a functional diagram of an exemplary system in accordance witha second exemplary embodiment of the present invention;

FIG. 5 is a functional diagram of an exemplary system in accordance witha third exemplary embodiment of the present invention;

FIG. 6 is a functional diagram of an exemplary system in accordance witha fourth exemplary embodiment of the present invention;

FIG. 7 is a functional diagram of an exemplary system in accordance witha fifth exemplary embodiment of the present invention;

FIG. 8 is a functional diagram of an exemplary system in accordance witha sixth exemplary embodiment of the present invention;

FIG. 9 is a functional diagram of an exemplary system in accordance witha seventh exemplary embodiment of the present invention;

FIG. 10 is a functional diagram of an exemplary system in accordancewith a eighth exemplary embodiment of the present invention;

FIG. 11 is a graph plotting image measurement against scene energy forvarious digital micromirror device levels;

FIG. 12 is an illustrative diagram showing the mapping between animaging sensor and a digital micromirror device;

FIG. 13 is a flow diagram showing the primary pixel control technique inaccordance with an exemplary embodiment of the present invention;

FIG. 14 is a graph plotting image measurement against scene energy forthe technique of FIG. 13;

FIG. 15 is a flow diagram showing the continuos pixel control techniquein accordance with an exemplary embodiment of the present invention;

FIG. 16 is a flow diagram showing a searching control in accordance withan exemplary embodiment of the present invention;

FIG. 17 is a flow diagram for controlling an intensity reweightingimaging in accordance with an exemplary embodiment of the presentinvention;

FIG. 18 is a flow diagram for controlling a spatially varying beamattenuator in accordance with an exemplary embodiment of the presentinvention;

FIG. 19 is a flow diagram for controlling an unfocused image inaccordance with an exemplary embodiment of the present invention;

FIG. 20 is a graph plotting radial distance against brightness and acompensation control signal;

FIG. 21 is an illustrative diagram showing filters in accordance with anexemplary embodiment of the present invention; and

FIG. 22 is an illustrative diagram showing filters in accordance with anexemplary embodiment of the present invention.

Throughout the Figs., the same reference numerals and characters, unlessotherwise stated, are used to denote like features, elements, componentsor portions of the illustrated embodiments. Moreover, while the presentinvention will now be described in detail with reference to the Figs.,it is done so in connection with the illustrative embodiments.

DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

Referring to FIG. 1, an exemplary system 100 for adaptive imagingenabling high quality imaging using a relatively low quality imagedetector is shown. A scene consisting of a very bright object 101, avery dark object 102, a bright object 103 and a dark object 104 is to berecorded. The exemplary system includes an image focusing device 110, adigital light processing apparatus 120, a relay lens 130, an imagedetector 140, and an image data processor 150 for processing image dataand providing feedback to a digital light processing apparatuscontroller 121.

The image focusing device 110 is positioned to focus the image of thescene on to the a digital light processing apparatus 120. The device maybe a lens, such as off the shelf imaging lens suitable for use with aconventional camera and of a focal length suitable for use with thedesired field of view and the optics of the digital light processingapparatus 120. The relay lens 130 is positioned to receive one of twopossible reflected images from the digital light processing apparatus120 and to focus that image onto image detector 140. The lens 130 mayalso be an the shelf imaging lens, should be of a focal length suitablefor use with the optics of the digital light processing apparatus 120and image detector 140.

The type of imaging and relay lenses used are not critical to thepresent invention, and thus other commercially available or custom builtlenses may be used. In addition, the lenses may be replaced withalternative image formation mechanisms such as pinhole, or omittedentirely. Likewise, while the image detector 140 used in the exemplaryembodiment is an eight-bit CCD, other image detectors, such as CMOSdetector, a microbolometer, or even a photomultiplier may be employed.

As shown in FIG. 1, the image data processor 150 is used to processimage data and provide feedback to the controller 121. The processor maybe implemented in either hardware or software, as further describedbelow. In either case, the captured image 151 is analyzed for quality152 to generate control image data 153 which is fed to the digital lightprocessing apparatus to modulate the intensity of light falling on thedetector 140. A second captured image 151 and the previously appliedcontrol image data 153 are then used to compute a high quality image 154of the scene.

In the example shown in FIG. 1, a scene is expected to have a very highdynamic range in terms of radiance values, such as including a verybright and dark objects 101, 102, and other bring and dark objects 103,104. An initially captured image can be analyzed by appropriate software150 to detect and dark intensity pixels corresponding to scene pointsthat lie outside the dynamic range of the image detector 140. Theradiance from the corresponding scene points is then modulated using adigital light processing apparatus 120 to ensure that they produce imagemeasurements that lie within the measurable range of the detector 140.Then, the captured image and the control image are used to compute afinal image that has an extended dynamic range.

In a preferred embodiment, the digital light processing apparatus 120 isa commercially available digital micromirror device (“DMD”). Referringto FIG. 2, the operation of a DMD 200 will be explained. A very largenumber of movable micromirrors 201, 202 reside on the surface 210 of theDMD. Each micromirror 101, 102 is able to switch at a rapid frequencybetween two tile angles (shown as φ) on either side of the optical axis220. The ratio of the time periods a DMD micromirror is at the two tiltangles is proportional to the intensity of the control signal applied tothat micromirrror. Therefore, the modulation of the light reflected ineach of the two directions if a function of the control signal. In FIG.2, a scene point 250 is imaged onto a single micromirror 203. Themicromirror 203 serves as the location of a virtual scene point whoseintensity is determined as the product of the radiance of the scenepoint and the modulation level for the micromirror. The complete DMDtherefore acts like a mirror with controllable local reflectivity.

A DMD is only one of several possible apparatus for digitally modulatingthe imaged light energy. Other digital light processing apparatus whichutilize controllable reflective surfaces to modulate an incident lightsource, e.g., by reflecting light in more than one direction or byreducing the energy of reflected light, may be utilized.

Referring next to FIG. 3, an embodiment 300 suitable for low resolutionspatial image modulation will be explained. When a DMD is placed infront of an image, it can be used to modulate the imaged light rays. Inthis case, given a finite imaging aperture, a neighborhood ofmicromirrors 301, 302, 303 (rather than a single one) modulate theradiance of a scene point before it is imaged by a pixel 312 on thedetector 140. Therefore, the irradiance of the pixel can be controlledby controlling a set of micromirrors, rather than a single one. In thiscase, the effective modulation function ends up being smooth or blurred,and the irradiance at any given pixel 311 cannot be controlled withouteffecting the irradiances of neighboring pixels.

In many applications smooth modulation functions will suffice and hencethis configuration is a useful one. For instance, when the modulator isused to compensate for smoothly radiometric effects such as vignetting,low-resolution modulation is sufficient. Even in applications such asHDR imaging, significant improvements in image quality can be achievedby adjusting the effective exposures of local neighborhoods of pixelsrather than individual pixels.

As shown in FIG. 4, an alternative arrangement 400 suitable for lowresolution spatial image modulation will be explained. In FIG. 4, a DMD200 is placed between the imaging lens 110 and the image detector 140.In this case, the DMD 200 serves to modulate as well as redirect theimaged light rays towards the image detector, again using a neighborhoodof micromirrors 401, 402, 403 corresponding to a scene point. Again,only smooth modulation signals can be achieved with this embodiment.

Referring next to FIG. 5, an preferred embodiment 500 suitable for highresolution spatial image modulation will be explained. In thisembodiment, the scene is imaged onto the DMD 200, where the DMD plane isfrontal, i.e., perpendicular to the optical axis of the imaging lens.When the micromirrors 501, 502, n, are located on a frontal plane, theyare all oriented at one of the two possible tilt angles with respect tothe plane. Each scene point is focused onto a single micromirror. Forexample, the light cone behind the lens 110 corresponding to a scenepoint 515 is reflected by the corresponding micromirror 505. Therefore,the micromirror serves as a virtual scene point that is located on theDMD plane. A relay lens 510 is then used to re-image the virtual scenepoint that is located on the DMD plane onto a single pixel on the imagedetector 140, e.g., the virtual scene point located on mircomirror 505is re-imaged onto pixel 525. Since the light cones are reflected by theDMD 200 at an angle with respect to the normal of the plane of the DMD,the image detector must be titled with respect to the optical axis ofthe re-imaging lens to obtain a focused imaged. The tile angle of imagedetector is easily determined from the focal length of the re-imaginglens and the location of the DMD plane using the Schiemflug condition.

In the embodiment shown in FIG. 5, the light energy received by eachpixel on the detector 140 can be modulated without effecting the lightreceived by neighboring pixels. Therefore, the modulation function usedfor a scene can be of a resolution that is matched with the resolutionof the captured image.

In FIG. 5, the DMD 200 may be of a significantly higher resolution thanthe image detector 140. In such cases, advanced filtering and processingfunctions can be performed by software 150 at an intra-pixel level toachieve super resolution spatial image modulation. Such techniques willbe addressed below.

Referring next to FIG. 6, an embodiment 600 suitable for non-homogeneousimage sensing will be explained. The embodiment is identical to thatshown in FIG. 5, except that the image detector 140 is replaced bydetector 610 adapted for a non-homogeneous pixel sensitivity to thevarious characteristics of light, e.g., color, spectrum brightness, orpolarization. For example, a detector with a Bayer color mosaic may beused. Since the dynamic range of each pixel can controlledindependently, the non-homogeneous nature of the detector is not anissue. The captured high quality data can be used to compute a highdynamic range color image.

Such non-homogeneous sensing is not restricted to color; the pixels mayhave different brightness sensitivities, e.g., as disclosed in S. K.Nayar et al., “High dynamic range imaging: Spatially varying Pixelexposures,” Proc. of Computer Vision and Pattern Recognition '00 (2000),the contents of which are incorporated by reference herein, or differentpolarization sensitivities, e.g., as described in M. Ben-Ezra,“Segmentation with Invisible Keying Signal,” Proc. of CVPR, pp 1:32-37(2000), the contents of which are incorporated herein. Those skilled inthe art will appreciate that other non-homogeneous image sensors maylikewise be employed.

Referring next to FIG. 7, an alternative embodiment 700 using multipledetectors will be explained. In this embodiment, the DMD 200 modulateslight by reflecting incident light in two directions. The arrangement isidentical to that of FIG. 5, except that an additional lense 710 andimage detector 740 are included, this time along the second axis ofreflection from the DMD.

If a control image with uniform brightness is applied to the DMD 200(see FIG. 1), the images received by the two detectors will haveeffective exposure values that are determined by the brightness value ofthe control image. This allows simultaneously captured of two imageswith different effective exposures. These two images can then be usedmore effectively to compute a control signal that is sent to the DMDcontroller to achieve high quality image data. For instance, for a givenscene point, if one detector produces a saturated brightness value, thecorresponding brightness produced by the second detector will likely notbe saturated and hence can be used to compute the control signal. Theinitial control image does not have to be constant one. A variety ofmodulation patterns may be used depending on the goals of theapplication.

Alternatively, the arrangement of FIG. 7 may be employed without use ofa control signal by using the DMD 200 as a beam splitter. For instance,if a constant control image is provide, two detectors have differentexposure values that are uniform over the image. A weighted average ofthese two captured images will yield an image with greater dynamic rangethan either of the two detectors. In this beam-splitting configuration,additional filters may be introduced in the optical paths of the twodetectors. These could be spectral, neutral density, and polarizationfilters. Likewise, beam splitting using a single DMD is not restrictedto a spatially uniform control image. More sophisticated control imagesmay be used based on the needs of the application. Thus, the advantageof employing a DMD 200 as a beam-splitter in the arrangement of FIG. 7is that it is programmable and hence flexible.

In a similar manner, the arrangement of FIG. 5 may be modified somultiple image detectors are used to sense imaged based on one DMDreflection direction. For example, in the arrangement 800 show in FIG.8., a partially transmitting beam splitter 830 may be used to relay thereflected image from the lens 510 to two detectors, 840, 841, eachhaving a different radiance sensitivity, via optical filters 820, 821.Such an arrangement may be helpful in speeding up convergence of thecontrol signal applied to the DMD 200.

Referring next to FIG. 9, an alternative embodiment 900 using cascadedimage modulators will be explained. The arrangement is similar to FIG.5, except that the detector 140 is replaced by a second DMD 901, with animage detector 940 positioned to receive one reflection from the DMD 901through an additional lens 910. With this arrangement, incoming lightcan be modulated with greater precision that as shown in FIG. 5. Sinceeach DMD has a limited number of quantization levels with respect to itsmodulation of light, the use of multiple DMDs serves to increase thenumber of modulation quantization levels. In the case of high dynamicrange imaging, this results in a greater dynamic range when compared toa system that uses a single DMD.

Further alternative arrangements using cascaded image modulators may beconstructed, such as the system 1000 shown in FIG. 10. For example, theconfiguration shown in FIG. 9 can be further enhanced by using two imagedetectors 940, 1040 and two additional lenses 910, 1010, to image thelight rays reflected by the second DMD 901, along its two reflectiondirections:

Referring to FIG. 11, the control of the DMD 200 will next be explained.At the pixel level, DMD modulation enables deflection of scene energyeither into an image sensor or away from it. The amount of measuredenergy varies linearly with the amount of time the DMD is deflectinglight into the sensor, which may be referred to as the primaryactivation time (“PAT”) for the sensor. In the following disclosureconcerning DMD control, it should be understood that a DMD is capable ofbeing operated at N different levels, each level being responsive tolog₂N of received control information, and that the smallest resolvabletime unit is t. As shown in FIG. 11, a DMD level of one provides thelongest PAT, with increasing levels decreasing the PAT.

Commercially available DMD devices have switching times of 2 usec withstabilized switching at 15 usec. Thus, for a 30 frame-per-second (“fps”)cameras, current DMDs can theoretically provide scalings from 1 down to1/2200. DMD controllers usually provide 3 input channels with 8 bits perchannel for a total of 24 control bits, with higher bit controllersbeing developed. Each channel is Pulse Code Width Modulated to provide256 temporal levels per channel. As each channel adds to the PAT in alinear manner, 768 levels of scaling are available without the need tomodify existing 8 bits per channel devices.

In the foregoing, the number of levels of scaling shall be moregenerally referred to as N. Let φ be the scaling associated with thePAT. For the two image detector configurations, e.g., FIG. 8, the seconddetector will receive energy not directed at the primary detector, henceit will receive energy for 33 ms-PAT, and have scaling 1-φ.

Letting the scene energy at a point be E(u, v), in an idealized sensoradaptive high dynamic range, the associated value in the 8-bit imagerI(x, y) will be as in Eq. 1:I(x,y)=min(└r _(xy)(φ,E(u,v)┘),255)  (1)where r_(xy) is the pixel response function. This is idealized sinceequation (1) does not take into account noise or inter-pixel influences.In an ideal CCD sensor, the function r_(xy) is linear. For a non-linearresponse, a radiometric calibration may be performed to determiner_(xy). To simplify the discussion below, assume that r_(xy) is linearor has been linearized by mapping with r_(xy) ⁻¹.

Let v correspond to the scene energy which for level=N results in ameasurement equal to 255−1. This is the maximum meaningful measurementthe system can make. If a measurement I(x, y) can be made with controlsignal set at C(x, y), then if no saturation occurs, i.e. I(x, y)<255),then the associated scene energy s, satisfies Eq. 2:

$\begin{matrix}{{\frac{{I( {x,y} )} + 1}{255} \cdot \frac{v{\cdot {C( {x,y} )}}}{N}} \geq s \geq {\frac{I( {x,y} )}{255} \cdot \frac{v{\cdot {C( {x,y} )}}}{N}}} & (2)\end{matrix}$an approximation to the energy is O(x,y)=I(x,y)C(x,y), which has aquantization separation of

$\begin{matrix}\frac{v \cdot {C( {x,y} )}}{255 \cdot N} & (3)\end{matrix}$which bounds the accuracy of the measurement of its dynamic range.

Ignoring noise and issues introduced from non-linear response functions,the quantization separation can be decreased, and hence the measureresolution increased, if the control signal is reduced to the smallestvalue that does not result in saturation of the CCD. Accordingly, thecontrol algorithm has multiple goals, including (a) the production of animage on the CCD array that neither causes saturation nor non-measurable(zero) levels, (b) insuring convergence even in a rapidly changingscene, (c) allowing mixing of the control signal and a measured image toproduce an image with a larger dynamic level and minimum quantizationerror for each measurement. Additional goals may include (d) theproduction of a measurement image I(x, y) which is directly useful, i.e.it has large dynamic range but does not require combination with thecontrol signal for human interpretation, (e) for stationary pixels,enabling the combing of measured images to further decrease thequantization error in intensity, (f) the implementation of a userdesired intensity remapping function, (g) the ability to image “dark”objects even if some pixels saturate, and (h) ensuring that brightobjects are not saturated even if dark objects are somewhat poorlyquantized.

In order to control intensity, the mapping between the pixels in themodulation device and the pixels in the imaging sensor must bedetermined. This mapping can be determined by the precise manufacturingof the device, or after construction by a calibration process.

An exemplary calibration process is shown in FIG. 12, where an image ofa structured input is used to estimate mapping parameters. The mappingmay be approximated as an affine transform, a perspective transform, ora general point-to-point mapping. To determine the association, acalibration patterns 1200 is projected into the camera 140. For example,a binary stripe pattern in each of x and y directions may be used.Rotating over many such patterns one can “encode” the pixels location bythe when it measures light and dark patterns. For each pattern in theworld (or entering the optics), the DMD control could then be varied viaan orthogonal binary stripe pattern and images collected. With thisapproach the full mapping can be computed from the resulting log₂255·log₂ N images.

Note that at the pixel level, the mapping will not necessarily be abijection, as a high resolution sensor, having an imaging resolutiongreater than that of the DMD, could be employed. In such a case, asingle DMD pixel will influence many image pixels. The control of thesesystems is similar to that discussed, except that the maximum pixel inthe CCD that corresponds to the DMD will be the determining factor forthe control algorithm.

However, in the more typical scenario, multiple modulation pixels areassociated with an imaging pixel, and the calibration process will needto provide a relative weighting. In this case, it is likely that somemodulation pixels will also be associated with multiple image pixels,and the control algorithm would be modified to set the “shared” pixelsto a value half way between the sharing pixels' desired control signals,and any pixel's with influence whole within the imaging pixel would beincreased/decreased to adjust for this impact of the shared pixels. Inthe following discussion, it will be presumed that each control pixelC(x, y) is associated with a modulation pixel D(x, y), with theunderstanding that coordinate transforms could be applied if needed formore complex scenarios.

As indicated above, the control software 150 may implement either closedloop or open loop control over the imaging process. An advantage ofopen-loop control in that it does not require real-time feedback fromthe imaging sensor. One approach to open loop control is to obtain thedistortion map R(x, y), using models of what is to be corrected, e.g.variation in measured brightness due to lens effects such as vignetting.Each pixel in this map is the reciprocal of the expected attenuation inthe distortion. A more general approach is to compute, using acalibration phase, a distortion map. One example is to correct spatiallyvarying sensor response effects. This can be accomplished by providinguniform intensity input, e.g. from an integrating sphere, and sequencingthrough all control signal to determined the value for each pixel thatwill result in the desired “flat” image.

Given a control map from the calibration phase, the map can be appliedwithout run time feedback and without introducing any delay. Multiplemaps may be stored, allowing corrections, e.g. when a lens is changed,or to apply to different desired intensity levels. However, suchopen-loop control does not insure that the image does not containsaturating pixels.

Referring next to FIG. 13, a technique for adaptively controlling asingle imaging sensor with an image focused on a DMD will be explained.In the following, it is assumed that both the pixel level alignment isknown, and the response function is linearized. It should be noted thatthe final image is processed 154 from the product of the measured andcontrol images, i.e. O(x, y)=I(x, y)*C(x, y), where I(x, y)ε[0,255] andC(x, y)ε[0, N]. From this it should be clear that larger values of thecontrol signal C, for the same measured image brightness I(x,y), areassociated with brighter scene points. Also, for a fixed scenebrightness, larger values of the control signal C(x,y) produce smallermeasured values I(x,y), with C(x,y)=N producing the smallest measurablesignals. This implies that there are multiple control algorithms thatcould produce the same measured output. While an exemplary controlalgorithm will be described, any algorithm that produces an equivalentconfiguration would suffice.

The fundamental part of the control algorithm is the computation of ascale that will reduce the measured value I(x, y) below saturation. Themaximum measurable scene energy corresponds to setting the φ=T, or theDMD timing to level N, a measurement represented as I₁(x, y). If a pixelsaturates at this setting, then the corresponding scene point cannot beaccurately measured, and is not considered a true measurement.

As shown in FIG. 13, the primary control method is based on a three-stepimaging sequence, 1310, 1320, 1330. The method generates a valid outputat every time instance and hence is stable. An initial image is captured1310 with control signal C₁(x, y)=N, or φ=T, yielding a measurement ofI₁(x, y) which is output 1311. The associated scene energy, s, satisfies

$\begin{matrix}{{\frac{{I_{1}( {x,y} )} + 1}{255} \cdot \frac{v{\cdot {C_{1}( {x,y} )}}}{N}} \geq s \geq {\frac{I_{1}( {x,y} )}{255} \cdot {\frac{v{\cdot {C_{1}( {x,y} )}}}{N}.}}} & (4)\end{matrix}$which has a quantization separation of

$\begin{matrix}{\frac{v \cdot {C_{1}( {x,y} )}}{255 \cdot N}\frac{v}{255}} & (5)\end{matrix}$

If I₁(x, y)<255−1 then the measurement accuracy can be increased and asecond image is captured 1320 and output 1321 by setting the level ofthe control signal for the second measurement, C₂(x, y) to

$\begin{matrix}{{C_{2}( {x,y} )} = {{\lfloor \frac{{I_{1}( {x,y} )} \cdot {C_{1}( {x,y} )}}{255} \rfloor + 1} = {\lfloor \frac{{I_{1}( {x,y} )} \cdot N}{255} \rfloor + 1}}} & (6)\end{matrix}$which will result in a measurement I₂(x, y) that satisfies

$\begin{matrix}{{\frac{{I_{2}( {x,y} )} + 1}{255} \cdot \frac{v{\cdot {C_{2}( {x,y} )}}}{N}} \geq s \geq {\frac{I_{2}( {x,y} )}{255} \cdot \frac{v{\cdot {C_{2}( {x,y} )}}}{N}}} & (7)\end{matrix}$which has a quantization separation of

$\begin{matrix}{{\frac{v{\cdot {C_{2}( {x,y} )}}}{N} \approx \frac{v \cdot \frac{{I_{1}( {x,y} )} \cdot N}{255}}{255 \cdot N}} = \frac{v{\cdot {I_{1}( {x,y} )}}}{255 \cdot 255}} & (8)\end{matrix}$

If N=256, so that all available levels in the DMD are not being used,then with two levels one can acquire a high quality image for eachpixel. As an example let us assume I₁=11, in which case O₁=2816. Then

$C_{2} = {{\lfloor \frac{2816}{255} \rfloor + 1} = 12.}$Upon capture of the second image, which has the value, I₂=239, thenO₂=2868.

If N>255, then the possible control values for C₂(x, y) does not includeall possible control signal levels and hence the estimation can beimproved by setting the control signal according to the secondmeasurement to define control signal for a third measurement 1330 whichis output 1331 by setting

$\begin{matrix}{{C_{3}( {x,y} )} = {{\lfloor \frac{{I_{2}( {x,y} )} \cdot {C_{2}( {x,y} )}}{255} \rfloor + 1} \approx {\lfloor \frac{{I_{2}( {x,y} )} \cdot {I_{1}( {x,y} )} \cdot N}{255 \cdot 255} \rfloor + 1}}} & (9) \\{\mspace{79mu}{{\frac{{I_{3}( {x,y} )} + 1}{255} \cdot \frac{v{\cdot {C_{3}( {x,y} )}}}{N}} \geq s \geq {\frac{I_{3}( {x,y} )}{255} \cdot {\frac{ {v{\cdot ( {{C_{3}( {x,y} )} + 1} )}} )}{N}.}}}} & (10)\end{matrix}$which has a quantization separation of

$\begin{matrix}{\frac{v{\cdot {C_{3}( {x,y} )}}}{255 \cdot N} \approx \frac{v \cdot {I_{2}( {x,y} )} \cdot {I_{1}( {x,y} )}}{255 \cdot 255 \cdot 255}} & (11)\end{matrix}$

Referring to FIG. 14, an example of the three step process of FIG. 13,where N=768, is shown. Let the first measured image provide I₁=1, inwhich case O₁=768. Then

$C_{2} = {{\lfloor \frac{768}{255} \rfloor + 1} = 4.}$For the second image, which has the value I₂=180, then O₂=720, and

$C_{3} = {{\lfloor \frac{720}{255} \rfloor + 1} = 3.}$With this control signal, I₃=241 and may be used to compute an outputO₃=696.

This method can handle modulators with up to 64 k levels. As DMDcontrollers are currently anticipated to be limited to a maximum of afew thousand levels, a three stage primary control should be sufficient.If digital modulates are developed with N>255·255, then the possiblevalues for C₃ (x, y) would still not include all possible control valuesand another step would be needed. In general, this primary controlalgorithm requires K steps where K is the smallest integer such thatN<=(255)^(K).

If the underlying scene energy does not change, the method is stable asit is deterministic and has no feedback loop. But since the underlyingscene energy is presumed to be dynamic, the measurements need to betested. Thus, at 1325, if I_(k)(x, y)==255 then step 1310 is repeated.Thus the dynamic method is guaranteed to converge only if the sceneenergy does not continuously change by More than 255 levels within the 2or 3 time frames needed to complete the process (e.g. 100 ms). If itfluctuates with a high amplitude with a frequency above that rate, themethod may loop between steps 1310 and 1320. Note that while fluctuatingthe algorithm will still have an approximate brightness for each point,it is just that the accuracy of that measurement will fluctuate.

The method of FIG. 13 guarantees convergence to an intensity estimatewithin three steps. Unfortunately, that method requires commencing atstep 1310 for each individual pixel, at making up to three separatemeasurements 1310, 1320, 1330 for each such pixel. However, in manysituations the scene will not undergo fast or high dynamic rangevariations. that require commencing at the first step 1310 for everypixel. Instead, for most pixels in most images the temporal variationwill be small.

Referring next to FIG. 15, a highly preferred technique for continuouslyadapting control of a single imaging sensor will next be explained. Thetechnique is similar to that disclosed in connection with FIG. 13,except that the measurement at step 1330 is modified to become step1530, an additional saturation determination 1535 is made after thatthird measurement 1530, and the parameters used for the thirdmeasurement are repeated for additional pixels 1540 unless a saturatedpixel is determined 1545.

The method of FIG. 13 has the goal of finding the smallest controlvalues (which produce smallest quantization separation), and thereforetends to produce image values near saturation. To reduce the frequencyof saturation and thus the need to return to step 1310, step 1530 inFIG. 15 is modified as described in equation 12, by making thedenominator in the computation a value less than 255:

$\begin{matrix}{{C_{3}^{\prime}( {x,y} )} = {{\lfloor \frac{2 \cdot {I_{2}( {x,y} )} \cdot {C_{2}( {x,y} )}}{255} \rfloor + 1} \approx {\lfloor \frac{2 \cdot {I_{2}( {x,y} )} \cdot {I_{1}( {x,y} )} \cdot N}{255 \cdot 255} \rfloor + 1}}} & (12) \\{\mspace{79mu}{{\frac{{I_{3}( {x,y} )} + 1}{255} \cdot \frac{v \cdot {C_{3}^{\prime}( {x,y} )}}{N}} \geq s \geq {\frac{I_{3}( {x,y} )}{255} \cdot {\frac{ {v \cdot ( {{C_{3}^{\prime}( {x,y} )} + 1} )} )}{N}.}}}} & (13)\end{matrix}$which has a quantization separation of

$\begin{matrix}{\frac{v \cdot {C_{3}^{\prime}( {x,y} )}}{255 \cdot N} \approx \frac{v \cdot 2 \cdot {I_{2}( {x,y} )} \cdot {I_{1}( {x,y} )}}{255 \cdot 255 \cdot 255}} & (14)\end{matrix}$

The control method will converge to a value near that denominator, so ina compromise between frequency of resetting and the quantization error,the desired pixel measurement level is preferably set at 128. This is anexample tradeoff between frequency of searching due to saturation andreduced dynamic range resolution. Values other than 128 providealternative tradeoffs. In the initial step, or when saturation occurs,convergence speed is increased by using 255 as the initial denominator.Accordingly, rather than commencing at step 1310 for every pixel, thispreferred embodiment uses a continuously adapting algorithm that onlyreverts to step 1310 when a pixel saturates.

Note while this is nearly twice the level of the quantization error ofthe primary control algorithm, the value of I′₃ is now expected to benearly half that of I₃, and to stay well away from the saturation level.Saturation will occur only when brighter objects move into a pixels'coverage.

In the methods shown in FIGS. 13 and 15, the first step is set at thelargest level, and hence, obtains the darkest measurable image. Whilethis has the advantage that it avoids saturation and hence always yieldsan approximation, it has the disadvantage that the darker parts of thescene may yield poor initial approximations. An alternative is have ancontrol algorithm that starts with the maximum exposure allowed and thenadjusts the control levels for pixels that are too bright.

Referring next to FIG. 16, an alternative control technique for a systemwhich includes a single DMD and a single sensor, which utilizessearching control with saturation, will be explained. When a pixelsaturates, an optimal setting for the next control level cannot bedetermined. Hence, the method performs a binary search, doubling thelevel added to the control signal for a pixel until it finds a levelthat does not saturate. To keep the quantization levels as small aspossible, it proceeds to reduce the control signal via a search untilthe measured signal is approximately equal to 128.

As shown in FIG. 16, the control signal is initially set to 1, i.e.,level 1, a measurement taken 1610, an output 1611 is made. Next, if thepixel image has a measured value of less than 255, 1615, and less than128, 1616, then the image is too dark, the control signal is decreased1640 by one-half of its current state, another measurement is made 1630and an output generated 1631. If the pixel image is not saturated and isgreater than 128, than it is in the desired range, and no change is madeto the control signal prior to the next image measurement 1630. If thepixel in the image is saturated, the control signal is increased by itsstep size 1620, and a new measurement is made 1630.

The previous control techniques presume the desired output is simply themost accurate scene measurement available. There are applications wherethe user might desire to remap the intensity via some monotonicintensity function. Common examples are gamma correction, radiometriccorrection, and histogram equalization. The first two of these arepixelwise computations, while the latter is a function of all the pixelsin the image. While one could simply take the output image and remap thevalues as desired, by enforcing the remapping in the control algorithm,extra resolution in measurements is fostered.

Referring next to FIG. 17, another alternative control technique for asystem which includes a single DMD and a single sensor, which utilizesintensity remapping, will be explained. The technique is similar to thatexplained in connection with FIG. 15, except that it uses a secondarycontrol image to hold the desired remapping value, R(x, y) which scalesthe measurement to produce the output, thus modifying measurement steps1320, 1530, and 1540 to 1720, 1730, and 1740, and resulting in outputvalues 1721, 1731, 1741 O(x, y)=I(x, y)·C(x, y)·R(x, y). For radiometriccalibration or user-specified intensity correction, for example,vignetting correction, the map R can be precomputed and stored. Unlikethe open-loop control, this technique will both correct for radiometricdistortions and ensure that the resulting image does not saturate.

For the histogram equalization the reweighting map R(x, y) needs to berecomputed for each image. To do this let H be the global histogramcomputed from the output O, quantized into V bins H[0] . . . H[V], wherethe brightness boundaries, B[i] are defined such that each H[j] hasapproximately the same number of pixels in it. Then

$\begin{matrix}{{R_{h}( {x,y} )} = {{\frac{1}{k}\mspace{14mu}{where}\mspace{14mu}{O( {x,y} )}} \in {H\lbrack k\rbrack}}} & (15)\end{matrix}$

This scaling is applied to the control signal as shown in FIG. 17. Forthe histogram equalization this is a continuously adapting mapping and,as an inherently global remapping criterion, it may change when any partof the input changes.

The raw measurement image of the histogram equalization algorithm is notitself histogram equalized, the output image O(x,y)=I(x,y)*C(x,y), whichis equalized in the full dynamic range, 0−N*255. If it is desired tohave it normalized in the raw image space [0,255], this can beaccomplished by having scaling R_(h)(x, y) by 1/N. Given that a highdynamic range image is being computed, it may seem that softwareprocessing could produce the same result. However, the control willreduce the quantization error in the acquired data, before it is everdigitized. Note that histogram equalization has an added benefit that itcan produce an image I(x, y) that will, when viewed directly, haveenhanced dynamic range. By reading off the values in the control signal,the boundaries of this histogram are also determined.

In the control techniques described so far, the output image wascomputed from the product of the control signal and the measured images.While this produces the desired adaptive dynamic range and maximizes themeasurement resolution per pixel, it requires that the output becomputed in a digital sense and have a resolution beyond that of thestandard resolution of a low level CCD or traditional 8-bit per channeldisplay devices. To directly display the measurement in a humanunderstandable requires another computation, which is undesirable.

An alternative is to control the system such that the measured image isstill understandable while still maintaining the adaptive high dynamicrange. To do this some of the measurement resolution must be sacrificed.Note that any mapping that produces a measured image that is monotonicin the scene measurement will be directly interpretable to a humanviewer, including the histogram equalization mentioned in the previoussection. As those skilled in the art will appreciate, there are numerousavailable methods for compressing the dynamic range of a signal, such asgamma compression. Any such dynamic range compression function can beincorporated into the above approach.

For example, to compute an arbitrary gamma correction applied to themeasured range of intensity with a full automatic gain control we wouldset the control signal according to the following. First set k=thelargest computer intensity value (I(x,y)*C(x,y)) in the previous frame.Then:

$\begin{matrix}{{C( {x,u} )} = {\lfloor {( \frac{O( {x,y} )}{k} )^{\gamma} \cdot \frac{N}{255}} \rfloor + 1}} & (16)\end{matrix}$

While they are scaled to be visible (and ordered) in the 8 bit range,the full values still can be computed. The digital outputs have the samewide dynamic range of the other control algorithms, but enforcing theorder in the display significantly increases the quantizationseparations. It is still, however, far better quantization that could beobtained with a traditional camera.

Referring next to FIG. 18, a technique for implementing control of adual image sensor system, such as that described in connection with FIG.7, will be described. Open look control may be employed to control useof a sensor. While a constant image could be employed, it is preferableto use a spatially variable form of control, with the desired beamsplitting being provided by the spatial splitting function, a spatialmap S(x, y) 1810. For example, if S(x, y)=X, the measured image wouldhave the fraction of incident light sent to a first sensor linearlyvarying from 0-1 across the image. The values in the spatial map are inthe form of a desired percentage split, which is the fraction of thelight that will be delivered to each output 1811, 1812, O₁(x, y). Theoutput is then used to control the DMD.

In FIG. 18, the control algorithm shows the output images unsealed. Thisallows the raw 8-bit output on each channel. In this sensor the rawoutputs could easily be analog NTSC output. If it is desired to have aconstant variable beam splitter it would be trivial to a control whichmight be controlled by a knob. With a knob the analog position would bedigitized to V set the single global control value for S(x, y)=V.

A technique for implementing closed loop control of a dual sensor systemwith a focused DMD, such as shown in FIG. 7, will next be explained. Thedual sensor system has the advantage that two measurements are made ineach time step. Similar to the case in the technique described inconnection with FIG. 13, an initial image is captured with the controlsignal C₁(x, y)=N, except that now two images I₁(x, y) and I₂(x, y) aremeasured in the first step. If I₁(x, y)=255, then the measured scenebrightness is outside the range of the measuring device, and the sensorsaturates.

If I₂(x, y)<255 the associated scene energy, s, satisfies

$\begin{matrix}{{\frac{{I_{2}( {x,y} )} + 1}{255} \cdot \frac{v}{N}} \geq s \geq {\frac{I_{2}( {x,y} )}{255} \cdot \frac{v \cdot}{N}}} & (17)\end{matrix}$which has a quantization separation of

$\begin{matrix}\frac{v}{255 \cdot N} & (18)\end{matrix}$Note, that if I₁(x, y)==0 then I₂(x, y)<255 (presuming N>255 as it isexpected to be). If I₂(x, y)<255, then O₁=I₂(x, y), then an accurate aspossible measurement has been achieved with the instant configuration.Otherwise we set O₁=I₁·C₁(x, y), and the technique moves on to step 1320where another two images, I₃(x, y) and I₄(x, y) are measured aremeasured.

The level of the control signal is set for the second time interval,C₂(x, y) to

$\begin{matrix}{{C_{2}( {x,y} )} = {\lfloor \frac{{I_{1}( {x,y} )} \cdot {C_{1}( {x,y} )}}{255} \rfloor = \lfloor \frac{{I_{1}( {x,y} )} \cdot N}{255 \cdot 255} \rfloor}} & (19)\end{matrix}$which will result in measurement I₃(x, y) and I₄(x, y), where

$\begin{matrix}{{\frac{{I_{3}( {x,y} )} + 1}{255} \cdot \frac{v \cdot {C_{2}( {x,y} )}}{N}} \geq s \geq {\frac{I_{3}( {x,y} )}{255} \cdot \frac{v \cdot {C_{2}( {x,y} )}}{N}}} & (20)\end{matrix}$which has a quantization separation of

$\begin{matrix}{{\frac{v \cdot {C_{2}( {x,y} )}}{N} \approx \frac{v{\cdot \frac{{I_{1}( {x,y} )} \cdot N}{255}}}{255 \cdot N}} = {\frac{v \cdot {I_{1}( {x,y} )}}{255 \cdot 255}.}} & (21)\end{matrix}$

Similarly,

$\begin{matrix}{{\frac{{I_{4}( {x,y} )} + 1}{255} \cdot \frac{v \cdot ( {N - {C_{2}( {x,y} )}} )}{N}} \geq s \geq {\frac{I_{4}( {x,y} )}{255} \cdot \frac{v \cdot ( {N - {C_{2}( {x,y} )}} }{N}}} & (22)\end{matrix}$which has a quantization separation of

$\begin{matrix}{{\frac{v \cdot ( {N - {C_{2}( {x,y} )}} )}{N} \approx \frac{v{\cdot \frac{( {255 - {I_{1}( {x,y} )}} ) \cdot N}{255}}}{255 \cdot N}} = {\frac{v \cdot {I_{1}( {x,y} )}}{255 \cdot 255}.}} & (23)\end{matrix}$

By design of the control algorithm, if the scene has not changedbrightness between steps 1310 and 1320, then I₃(x, y)<255. If I₄(x,y)==255 then the output is O₂=I₃(x, y)·C₂ (x, y). If however I₄(x,y)<255, then the output is:

$\begin{matrix}{O_{2} = \lfloor \frac{{{I_{3}( {x,y} )} \cdot {C_{2}( {x,y} )}} + {{I_{4}( {x,y} )} \cdot ( {N - {C_{2}( {x,y} )}} )}}{2} \rfloor} & (24)\end{matrix}$This results in a two-step control algorithm, rather than the three-stepcontrol algorithm of FIG. 15.

As in the one chip case, this control technique can be directly appliedto a single chip color camera. Given pixel level alignment, the methodapplies directly; without alignment the image measurements must beinterpolated to provide approximations of the color measurements at eachposition before apply the control method can be applied.

Next, a more generalized technique for implementing control of an N CCDsensor system with a focused DMD will be explained. Control of a systemusing three or more sensors focused on a DMD is similar to control of asingle sensor system, except that care must be taken to insure that noimage is saturated. If the first image is captured using C₁(x, y)=N andmeasured, say I_(R1)(x, y), I_(G1)(x, y), I_(B1)(x, y) for an RGBsensor, then the control for the second image will be:I _(m1)(x,y)=max(I _(R1)(x,y),I _(G1)(x,y)I _(B1)(x,y))  (25)with C₂(x, y) set to

$\begin{matrix}{{C_{2}( {x,y} )} = {{\lfloor \frac{{I_{m\; 1}( {x,y} )} \cdot {C_{1}( {x,y} )}}{255} \rfloor + 1} = {\lfloor \frac{{I_{m\; 1}( {x,y} )} \cdot N}{255} \rfloor + 1}}} & (26)\end{matrix}$

In a similar manner,I _(m2)(x,y)=max(I _(R2)(x,y),I _(G2)(x,y)I _(B2)(x,y))  (27)can make the final measurement. This approach of computing a pixel-wisemaximum of the input channels and using it to control the next levelalso applies to the sensor where there are CCD sensors on both sides ofthe DMD, as in FIG. 7, with the understood modifications to equation 19above.

Referring next to FIG. 19, a technique for implementing control of asingle CCD sensor with an unfocused image on a DMD will be explained.The technique is similar to that explained in connection with FIG. 15,except that it uses a high resolution secondary control signal whichscales the measurement to produce the output, thus modifying measurementsteps 1320, 1530, and 1540 to 1920, 1930, and 1940, and resulting inoutput values 1921, 1931, 1941 O(x, y)=I(x, y)·C(x, y)·R(x, y). With anunfocused image on the DMD, the brightness of each pixel can not beindependently controlled via direct means. Because the image isdefocused, the energy falling on one mirror facet will not necessarilyend up on a single pixel on the sensor, as the bundle of rays willspread out over neighboring pixels.

As shown in FIG. 19, this control is based on first computing a highresolution control signal as in the focused case, and then the controlsignal is de-convolved with a kernel k. The choice of the convolutionkernel k (both fall off and width) is a function of the level ofdefocusing expected on the DMD.

Next, the problem of quantization error will be addressed. In almost anyof the above control techniques, there is a range of settings such thatthe measured intensity image will not be not saturated. Each settingwill yield an approximation of the scene energy. While most of thecontrol algorithms attempt to produce as accurate an approximation aspossible from a single measured frame, it is also possible, for scenepoints that do not change quickly, to obtain even better estimates byusing a control algorithm that uses multiple different control values.

In a standard camera, temporal averaging cannot decrease thequantization separation (though it can reduce camera noise) because thescaling of the scene energy is fixed. However, most of theabove-described control algorithms can provide different scalings,yielding multiple approximations. An improved algorithm could combinethese results to provide a more accurate estimate. In a noiseless case,multiple approximations can be combined:

$\begin{matrix}\begin{matrix}{O_{1},{{l = {{I_{1}( {x,y} )} \cdot {C_{1}( {x,y} )}}};}} & {O_{1},{r = {( {{I_{1}( {x,y} )} + 1} ) \cdot {C_{1}( {x,y} )}}}} \\{O_{2},{{l = {{I_{2}( {x,y} )} \cdot {C_{2}( {x,y} )}}};}} & {O_{2},{r = {( {{I_{2}( {x,y} )} + 1} ) \cdot {C_{2}( {x,y} )}}}} \\\vdots & \vdots \\{O_{p},{{l = {{I_{p}( {x,y} )} \cdot {C_{p}( {x,y} )}}};}} & {O_{p},{r = {( {{I_{p}( {x,y} )} + 1} ) \cdot {C_{p}( {x,y} )}}}} \\{O^{\prime} = {\frac{1}{2}( {{\max_{i}O_{i}},{l + {\min_{i}O_{i}}},r} )}} & \;\end{matrix} & (28)\end{matrix}$

For example, presuming the measurement of a point whose true sceneenergy should be represented as 24311, requiring 16 bits forrepresentation, but only 8 bit sensors are used which claim their valuedownward, control signals of 112, 128, 168 could be used and for whichthat ideal sensor would return measurements of 217, 189, 172.

$\begin{matrix}\begin{matrix}{O_{1},{{l = {{217*112} = 24303}};}} & {O_{1},{r = {{218*112} = 24416}}} \\{O_{2},{{l = {{189*128} = 24192}};}} & {O_{2},{r = {{190*128} = 24320}}} \\{O_{3},{{l = {{144*168} = 24192}};}} & {O_{3},{r = {{145*168} = 24360}}} \\{{O^{\prime} = {\frac{1}{2}{\max_{i}O_{i}}}},{l + {\min_{i}O_{i}}},r} & {= {{\frac{1}{2}( {24303 + 24320} )} = 24311.50}}\end{matrix} & (29)\end{matrix}$which has an error of only 0.5, whereas taking ½O_(i),l+O_(i),r yields aerror of at 35 for each individual measurement. In the more generalnoisy measurement case, someone skilled in the art will recognize thatthere are various algorithms for combining multiple noisyapproximations, e.g. a mean weighted by the expected error given theimaging sensor's noise model.

The control techniques described above may be implemented in eitherhardware or software. For several reasons unrelated to the techniquesthemselves, e.g. to ensure a sharp CCD image, it may be advantageous forthe sensor system to be linked to a digital signal processor (“DSP”),and thus such a hardware platform may be preferable. With the exceptionof the control flow for unfocused images and the control for histogramequalization, the remaining control flow techniques are all pointwiseindependent allowing them to computed as the DSP processes in measuredimage in scan line order. For unfocused system the system would need tobuffer sufficient lines to implement the convolution. For the histogramequalization it would need to store the histogram in a buffer. In eachcase it is anticipated that the control signal C(x, y) would be storeddigitally and used to provide the control signal to the DMB. By usingdual ported memory this control signal could be read by the DMDcontroller while a new control signal is being computed. Since thesignal is computed pixel-wise, the timing of these updated would notneed to be synchronized.

Alternatively, the control techniques may be realized in software on anexternal computation platform, e.g. a PC connected to a frame-grabberthat is providing the intensity measurements. The control signal wouldthen be provided to the DMD by considering the control signal as a“display” signal and using the existing circuit developed for DMDcontrol. such an implementation, the signal should be encoded using thefirst 8 bits of the control signal encoded as the desired intensity forthe green channel G, with red and blue channel either 0 or 255,depending on highest two bits of the control signal.

Referring next to FIGS. 20-22, image compensation and filtering will beexplained. Spatial image modulation has strong implications for imageprocessing in general. Thus far, only adaptive imaging techniques thatuse a feedback control loop (see FIG. 1) to enable the imaging system toadapt to scene appearance have been addressed. However, many usefuloperations can also be achieved using an open loop system.

For instance, a captured image is often compensated for undesirableradiometric effects. Examples of such effects are the “cosine to thepower of four” fall-off found in many imaging systems, brightness decaytowards the edge of the field of view due to lens vignetting, variationsin pixel sensitivities due to defects, etc. In such cases, a DMDmodulator can be given a “compensation” control image that results inoptical compensation for the above effects.

The radiometric fall-off function and the corresponding control imageare illustrated in FIG. 20. In other cases, for visual monitoringpurposes, it may be useful to enhance (highlight) certain regions of theimage. Here again, appropriate controls images can be used to accomplishthis in real-time without the need for computations. Also, compensationschemes can also be made adaptive by incorporating them into closed-loopcontrol. For instance, if moving regions in scene need to be highlight,a simple motion detector can be applied to the captured image and theregions of interest can be modulated appropriately.

Other implications exist for image processing. The most common operationin image processing is the convolution of the image using a “mask” or a“filter.” Here again, a significant portion of the computations involvedcan be done optically using spatial modulation.

As an example, consider the popular Laplacian filter kernel 2110 shownin FIG. 21. This operator is often used to find edges and extract thehigh frequency components in images. If the control image 2120 shown inFIG. 21 is applied to a DMD, each scene radiance gets multiplied by thecorresponding control image value. By applying the (3×3) computationalkernel 2130 shown to non-overlapping 3×3 windows in the captured image,the Laplacian value for the pixel at the center of each of these 3×3windows is obtained. Such an approach has significant computationalbenefits; the computational kernel shown on the right only includes 1'sand −1's. Therefore, the result is obtained by simply addition andsubtraction operations applied to the captured image. That is, all themultiplications (which are significantly more computationally expensivethan additions and subtractions) are done optically by the modulator.

As another example, in the system of FIG. 5, the DMD may be of asignificantly higher resolution than the image detector. In such cases,interesting filtering/processing functions can be performed at anintra-pixel level, as illustrated in FIG. 22.

For instance, the spatial response function of each pixel can be varied.Pixels typically serve as buckets that collect photons. Intra-pixelcontrol therefore enables control of the light gathering properties ofthese buckets. For instance, one may apply a derivative filter, such asthe Laplacian filter, to the light cone. Since such operations typicallyinvolve applying kernels that have both positive and negative elements,the kernel may be implemented using two optical kernels 2220, 2230 thatare both positive (one with the positive elements of the original kerneland zeroes everywhere else, and the other with the negative elements ofthe original kernel and zeroes everywhere else). The two pixelmeasurements obtained by applied the two optical kernels are thensubtracted to obtain the final result. This final result has aresolution that cannot be achieved using traditional digital imageprocessing methods. Any computational filter has a finite supportassociated with it (it spans a neighborhood in the image). This severelylimits the resolution of the computed result. In the implementationshown in FIG. 22, the support width of kernel is a single pixel.Therefore, the computed kernel output has very high spatial resolution.

This approach can also be used to alter the spatial sensitivitycharacteristics of individual pixels on the image detector. Forinstance, by using a simple 3×3 Gaussian-like filter one can shape thespatial sensitivity of each pixel to be more Gaussian like (greatersensitivity to light falling in the center of the pixel).

Note that these super-resolution filtering operations can benefit fromthe multiple detector configuration shown in FIG. 7; in this case, thetwo detectors simultaneously capture images that correspond to theresults of applying a positive filter f(x, y) and its complementaryfilter 1−f(x, y). Therefore f(x, y) may be selecting such that it andits complement can be used together to realize more interesting filters.

The foregoing merely illustrates the principles of the invention.Various modifications and alterations to the described embodiments willbe apparent to those skilled in the art in view of the teachings herein.For example, while the above description assumes imaging in the visiblelight spectrum domain, the adaptive imaging methods are directlyapplicable to imaging of any form electromagnetic radiation, includingradar, infra-red, near infra-red, and X-ray. It will thus be appreciatedthat those skilled in the art will be able to devise numerous systemsand methods which, although not explicitly shown or described herein,embody the principles of the invention and are thus within the spiritand scope of the invention.

The invention claimed is:
 1. A system for adaptively imaging a scene,comprising: a digital light processing apparatus adapted forcontrollably reflecting at least first and second images of said scenein at least a first direction based on control data to thereby generatecorresponding first and second intensity modulated images of said scene;an image detector, optically coupled to said digital light processingapparatus along said first direction of reflection, for detecting saidfirst and second intensity modulated images corresponding to said atleast first and second images of said scene and generating first andsecond detected image data corresponding to said at least first andsecond images; and an image data processor, coupled to said imagedetector and receiving said first and second detected image datatherefrom, and coupled to said digital light processing apparatus, forprocessing said first detected image data into first control data,providing said first control data to said digital light processingapparatus for adaptive control thereof, and generating output image datafrom said second detected image data and said first control data, saidoutput image data having a quality greater than said first detectedimage data or said second detected image data.
 2. The system of claim 1,wherein said digital light processing apparatus comprises a digitalmicromirror device.
 3. The system of claim 2, further comprising animage focusing device, optically coupled to said scene and said digitalmicromirror device and positioned therebetween, for focusing said atleast first and second images of said scene onto said digital lightprocessing apparatus.
 4. The system of claim 3, wherein said imagefocusing device comprises a lens.
 5. The system of claim 2, furthercomprising an image focusing device, optically coupled to said imagedetector and said digital micromirror device and positionedtherebetween, for focusing said first and second intensity modulatedimages of said scene onto said image detector.
 6. The system of claim 5,wherein said image focusing device comprises a lens.
 7. The system ofclaim 6, wherein said image detector comprises a CCD camera.
 8. Thesystem of claim 2, wherein said image data processor comprises a generalpurpose computer including software for processing said first detectedimage data into said first control data; providing said first controldata to said digital micromirror device, and generating said outputimage data from said second detected image data and said first controldata.
 9. The system of claim 1, further comprising a second imagedetector, optically coupled to said digital light processing apparatusalong a second direction of reflection therefrom and adapted forreceiving further intensity modulated images of said scene therefrom,for detecting said further intensity modulated images of said scene andgenerating corresponding further detected image data, and wherein saidimage data processor is further coupled to said second image detectorfor receiving said further detected image data therefrom, and adaptedfor processing said second detected image data into second control data.10. The system of claim 1, further comprising a beam splitter, opticallycoupled to said digital light processing apparatus along said firstdirection of reflection, for splitting said intensity modulated imagestherefrom into a partial transmission and a partial reflection, whereinsaid image detector is optically coupled to said beam splitter along adirection of said partial transmission, and a second image detector isoptically coupled to said beam splitter along a direction of saidpartial reflection for detecting said intensity modulated images of saidscene along said direction of partial reflection and generatingcorresponding detected image data, and wherein said image data processoris further coupled to said second image detector for receiving saidcorresponding detected image data therefrom, and adapted for processingsaid corresponding detected image data into second control data.
 11. Thesystem of claim 1, further comprising a second digital light processingapparatus adapted for controllably reflecting said first and secondintensity modulated images of said scene in at least a second direction,optically coupled to said digital light processing apparatus along saidfirst direction of reflection thereof, wherein said image detector isoptically coupled to said second digital light processing apparatusthrough said second direction of reflection from said second digitallight processing apparatus.
 12. The system of claim 11, furthercomprising a second image detector, wherein said second image detectoris optically coupled to said second digital light processing apparatusthrough a further direction of reflection from said second digital lightprocessing apparatus.
 13. A system for adaptively imaging a scene,comprising: a digital light processing apparatus adapted forcontrollably reflecting at least first and second images of said scenein at least a first direction based on control data to thereby generatecorresponding first and second intensity modulated images of said scene;an image focusing device, optically coupled to said scene and saiddigital light processing apparatus and positioned therebetween, forfocusing said first and second images of said scene onto said digitallight processing apparatus; an image detector, optically coupled to saiddigital light processing apparatus along said first direction ofreflection, for detecting said first and second intensity modulatedimages of said scene and generating first and second detected imagedata; and an image data processor, coupled to said image detector andreceiving said first and second detected image data therefrom, andcoupled to said digital light processing apparatus, for processing saidfirst detected image data into first control data, providing said firstcontrol data to said digital light processing apparatus for adaptivecontrol thereof, generating output image data from said second detectedimage data and said first control data, and processing said output imagedata with filter data to generate final image data.
 14. The system ofclaim 13, wherein said controllable digital light processing apparatuscomprises a digital micromirror device.
 15. The system of claim 14,wherein said image focusing device comprises a lens.
 16. The system ofclaim 14, further comprising a second image focusing device, opticallycoupled to said image detector and said digital micromirror device andpositioned therebetween, for focusing said first and second intensitymodulated images of said scene onto said image detector.
 17. The systemof claim 16, wherein said second image focusing device comprises a lens.18. The system of claim 17, wherein said image detector comprises a CCDcamera.
 19. The system of claim 13, wherein said filter data comprisesLaplacian filter data.
 20. The system of claim 13, wherein said filterdata comprises first and second optical kernel data.
 21. A method forcontrolling adaptive imaging of a scene using an N-level digital lightprocessing apparatus adapted for controllably reflecting at least firstand second images of said scene in at least a first direction to therebygenerate corresponding intensity modulated images of said scene along atleast said first direction, comprising: setting a digital lightprocessing apparatus control signal corresponding to said scene to aninitial level N; taking a first measurement of scene energy at one ormore pixels corresponding to at least a portion of said scene using saidinitial control signal; revising said digital light processing apparatuscontrol signal to a second level using said first measurement and asecondary control value, the secondary control value based at least inpart on a level of defocusing of the digital light processing apparatus;taking a second measurement of scene energy at said one or more pixelsusing said revised control signal; and generating an output image usingsaid second measurement of scene energy and said revised control signal,said output image having a quality greater than the first measurement ofscene energy or the second measurement of scene energy.
 22. The methodof claim 21, further comprising: comparing said second measurement witha threshold value; updating said digital light processing apparatuscontrol signal to a third level based on said comparison of said secondmeasurement; taking a third measurement of scene energy at said one ormore pixels using said updated control signal; and generating saidoutput image using said third measurement of scene energy and saidupdated control signal, said output image having a quality greater thanthe second measurement of scene energy or the third measurement of sceneenergy.
 23. The method of claim 22, wherein taking said measurements ofscene energy at said one or more pixels comprises taking saidmeasurements of scene energy at a first group of one or more pixels anda second group of one or more pixels, said second group comprising oneor more pixels corresponding to a different portion of said scene fromsaid first group.
 24. The method of claim 23, further comprising:comparing any most recent measurements with a threshold value; updatingsaid digital light processing apparatus control signals to one or morenew levels using said comparisons of said any most recent measurements;taking additional measurements of scene energy at said first and secondgroups of one or more pixels using said updated control signals updatedusing said comparisons of said any most recent measurements; andgenerating said output image using said additional measurements of sceneenergy and said updated control signals, said output image having aquality greater than said additional measurements of scene energy. 25.The method of claim 21, further comprising determining whether saidfirst measurement is below a first threshold value and if said firstmeasurement is below said threshold value, determining whether saidfirst measurement is below a second threshold value; and whereinrevising said digital light processing apparatus control signalcomprises revising said digital light processing apparatus controlsignal to a lower second level if said first measurement is determinedto not be below said first threshold value, to a higher second level ifsaid first measurement is determined to be below said first thresholdvalue and below said second threshold value, and to the same secondlevel if said first measurement is determined to be below said firstthreshold value and not below said second threshold value.
 26. Themethod of claim 21, wherein revising said digital light processingapparatus control signal comprises revising said digital lightprocessing apparatus control signal to the second level using said firstmeasurement and a remapping value.
 27. The method of claim 26, whereinsaid one or more pixels comprises a first group of one or more pixelsand a second group of one or more pixels, said second group comprisingone or more pixels corresponding to a different portion of said scene assaid first group, the method further comprising: comparing said secondmeasurement to a threshold value; updating said digital light processingapparatus control signal to a third level using said remapping value andsaid comparison of said second measurement; taking a third measurementof scene energy at said one or more pixels using said updated controlsignal; comparing any most recent measurements with a further thresholdvalue; updating said digital light processing apparatus control signalsto one or more new levels using said remapping value and saidcomparisons of said any most recent measurements; taking additionalmeasurements of scene energy at said two or more groups of one or morepixels using said updated control signals updated using said comparisonsof said any most recent measurements; and generating said output imageusing said additional measurements of scene energy and said updatedcontrol signals, said output image having a quality greater than saidadditional measurements of scene energy.
 28. The method of claim 21,wherein said one or more pixels comprises a first group of one or morepixels and a second group of one or more pixels, said second groupcomprising one or more pixels corresponding to a different portion ofsaid scene as said first group, the method further comprising: comparingsaid second measurement to a threshold value; updating said digitallight processing apparatus control signal to a third level using saidsecondary control value and said comparison of said second measurement;taking a third measurement of scene energy at said one or more pixelsusing said updated control signal; comparing any most recentmeasurements with a further threshold value; updating said digital lightprocessing apparatus control signals to one or more new levels usingsaid secondary control value and said comparisons of any most recentmeasurements; taking additional measurements of scene energy at saidfirst and second groups of one or more pixels using said updated controlsignals updated using said comparisons of said any most recentmeasurements; and generating said output image using said additionalmeasurements of scene energy and said updated control signals, saidoutput image having a quality greater than said additional measurementsof scene energy.
 29. The method of claim 21, wherein said firstmeasurement includes first measurements from at least two differentlypositioned sensors and said second measurement includes secondmeasurements from said at least two differently positioned sensors. 30.The method of claim 21, wherein said initial digital light processingapparatus control signal includes two or more control signals and saidrevised digital light processing apparatus control signal includes twoor more control signals.
 31. A method for controlling adaptive imagingof a scene using an N-level digital light processing apparatus adaptedfor controllably reflecting an image of said scene in at least a firstdirection to thereby generate corresponding intensity modulated imagesof said scene, comprising: setting a digital light processing apparatuscontrol signal corresponding to a portion of said scene to an initiallevel N; taking a measurement of scene energy at one or more pointscorresponding to said scene using said initial control signal;convolving said measurement with filter data, said filter datacomprising Laplacian filter data; and revising said digital lightprocessing apparatus control signal to a second level using saidconvolution of said measurement and said filter data.
 32. A method forcontrolling adaptive imaging of a scene using an N-level digital lightprocessing apparatus adapted for controllably reflecting an image ofsaid scene in at least a first direction to thereby generatecorresponding intensity modulated images of said scene, comprising:setting a digital light processing apparatus control signalcorresponding to a portion of said scene to an initial level N; taking ameasurement of scene energy at one or more points corresponding to saidscene using said initial control signal; convolving said measurementwith filter data; revising said digital light processing apparatuscontrol signal to a second level using said convolution of saidmeasurement and said filter data; convolving said measurement withsecond filter data; and determining a final image from said convolutionof said measurement with said filter data and said convolution of saidmeasurement with said second filter data.
 33. The method of claim 32,wherein said filter data comprises first optical kernel data and saidsecond filter data comprises second optical kernel data.